Computer  Science  Departmen 


TECHNICAL  REPORT 


iH 

• 

• 

• 

u 

• 

00  .., 

o 

^a 

4-1 

rH 

1    w 

E 

o;-^ 

dJ 

>l 

5^S 

w 

^  " 

fO 

o  ■-* 

•  • 

CJ^ 

Eh 

c 

W 

O   (0 

W 

< 

ASSET  :  A  System  to  Select  and  Evaluate  Tests 

Phyllis  G.  Frankl,  Stewart  N.  Weiss  and  Elaine  J.  Weyuker 

January  10,  1985 

Technical    Report    #148 


NEW  YORK  UNIVERSITY 


Department  of  Computer  Science 
Courant  Institute  of  Mathematical  Sciences 

251  MERCER  STREET,  NEW  YORK,  N.Y.  10012 


V 


_    i-'^ife  I?'- 


"fr     •*:■ 


ASSET  :  A  System  to  Select  and  Evaluate  Tests 

Phyllis  G.  Frankl,  Stewart  N.  Weiss  and  Elaine  J.  Weyuker 

January  10,  1985 
Technical    Report    #148 


ABSTRACT 

This  paper  describes  ASSET,  a  tool  which  uses  information  about  a 
program's  data  flow  to  aid  in  selecting  test  data  for  the  program  and  to 
evaluate  test  data  adequacy.  ASSET  is  based  on  the  family  of  data  flow  test 
selection  eind  test  data  adequacy  criteria  developed  by  Rapps  and  Weyuker. 
ASSET  accepts  as  input  a  program  written  in  a  subset  of  Pascal,  a  set  of  test 
data,  and  one  of  the  data  flow  adequacy  criterion  and  indicates  to  what  extent 
the  criterion  has  been  satisfied  by  the  test  data. 
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1.   Introduction 

The  selection  of  test  data  and  evaluation  of  its  adequacy  are  among  software 
engineering's  most  important  problems  today.  We  have  designed  and  implemented  a  tool, 
called  ASSET,  which  uses  data  flow  information  to  aid  in  the  selection  and  evaluation  of  test 
data  for  programs  written  in  a  small  subset  of  Pascal.  It  can  also  be  used  to  aid  in  debugging 
and  to  detect  data  flow  anomalies.  We  intend  to  use  this  tool  as  a  prototype  for  one  which 
performs  these  functions  for  programs  written  in  Pascal. 

The  goal  of  testing  is  to  detect  errors.  If  the  test  data  has  been  carefully  selected,  then 
its  failure  to  expose  errors  should  increase  one's  confidence  that  the  program  behaves 
according  to  its  specification.  The  program  is  run  on  a  subset  of  its  input  domain,  and  the 
output  is  compared  to  the  expected  result.  We  will  assume  the  availability  of  an  "oracle" 
which  can  determine  whether  or  not  a  given  input-output  pair  is  correct.  Since  nothing  short 
of  exhaustive  testing,  which  is  usually  infeasible,  can  guarantee  that  the  program  is  correct, 
research  in  testing  aims  at  finding  methods  of  producing  test  sets  which  are  small  enough  to 
be  useful  in  practice,  yet  which  are  likely  to  expose  existing  errors. 

There  are  two  aspects  to  this  problem:  the  selection  of  test  data  and  the  evaluation  of 
test  data  adequacy.  A  test  data  selection  method  is  a  procedure  for  choosing  elements  to 
include  in  the  test  set.  An  adequacy  criterion  is  a  rule  used  to  determine  whether  "enough" 
testing  has  been  done. 
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Several  soxirces  of  information  are  available  for  use  in  test  data  selection  and  in 
adequacy  determinations,  the  most  important  being  the  program  text  and  the  specification. 
Those  methods  which  are  based  on  the  program  text  are  called  program-based  and  those 
based  on  the  specification  are  called  specification-based.  Program-based  methods  have  the 
advantage  of  being  more  amenable  to  automation. 

A  number  of  program-based  test  data  selection  methods  and  adequacy  criteria  have 
been  proposed  and  tools  to  implement  them  have  been  built.  These  include  statement 
testing,  branch  testing,  and  path  testing  [Hua75,How76,How78,Mil74,Woo80].  All  of  these 
are  based  on  control  flow.  They  are  based  on  the  rationale  that  one  cannot  have  much 
confidence  that  the  program  is  correct  if  some  statement,  branch,  etc.  has  not  been  executed 
by  any  test  data. 

Rapps  and  Weyuker  [Rap80,Rap82]  have  proposed  a  family  of  test  data  selection  and 
adequacy  criteria  which  are  based  on  data  flow  analysis.  Similar  criteria  were  proposed  in 
[Nta84]  and  [Las83].  The  intuition  motivating  these  criteria  is  that  one  cannot  have  much 
confidence  that  the  program  is  correct  if  the  result  of  some  variable  definition  has  never  been 
used  during  testing.  These  criteria  are  program-based  and  have  been  shown  to  be  more 
powerful  than  branch  testing. 

ASSET  determines  whether  a  given  test  set  adequately  tests  a  given  program  with 
respect  to  the  test  data  adequacy  criteria  defined  in  [Rap80,Rap82].  In  section  two  we 
summarize  this  theory.  Section  three  describes  the  design  and  implementation  of  the  ASSET. 
Section  four  presents  a  number  of  applications. 

2.   Data  Flow  Testing 

A  family  of  test  data  selection  criteria,  each  based  on  the  program's  data  flow 
characteristics,  was  defined  in  [Rap80]  and  [Rap82]  for  a  very  simple  universal  programming 
language  consisting  of  assignment  statements,  conditional  and  unconditional  transfer 
statements,  and  input  and  output  statements.    This  language  has  the  same  structxire  and 
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seraantics  as  the  subset  of  Pascal  accepted  by  ASSET  (see  figure  1.)  We  now  present  a 
summary  of  the  terminology  used. 


<program>  ::=< program  heading>; 

<  label  declaration  part>; 

<  variable  declaration  part>; 

begin  <stateraent>{;<statement>}  end. 

<statement>  ::=  [<label>:]<siraple  statement> 

<  simple  statement>    ::=  <  assignment  statement> 

I  <  conditional  transfer  statement> 
I  <  unconditional  transfer  statement> 
I  <  input  statement> 
I  <  output  statement> 

< assignment  statement>  ::=  <identifier>  :-  <expression> 

<  conditional  tranfer  statement>        ::=  If  <expression>  then  goto  <label> 

< unconditional  transfer  statement>  ::=  goto  < label > 

< input  statement>     ::=  read[(<identlist>)] 

|readln[(<identlist>)] 

< output  statement>  ::=  write[(<exprlist>)] 

I  writein[(<exprlist>)] 


<identlist> 
<exprlist> 


::=  <identifier>  [,<identlist>] 
::=  <expression>  [,<exprlist>] 


where 

<  program  heading>,  <  label  declaration  part>, 

<  variable  declaration  part>,  <label>, 
<identifier>,  and  <expression> 

are  defined  as  in  Pascal. 


Figure  1.  Syntax  of  the  Language 

A  program  can  be  uniquely  decomposed  into  a  set  of  disjoint  blocks  of  ordered 
statements  having  the  property  that  whenever  the  first  statement  of  the  block  is  executed,  the 
other  statements  are  executed  in  the  given  order.  Furthermore,  the  first  statement  of  the 
block  is  the  only  statement  which  may  be  executed  directly  after  the  execution  of  a  statement 
in  another  block.  Intuitively,  a  block  is  a  chunk  of  sequential  code  which  is  always  executed 
as  a  unit.    The  program  is  represented  by  a  program  graph  in  which  the  nodes  correspond  to 
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the  blocks  of  the  program,  and  edges  indicate  possible  flow  of  control  between  blocks.  The 
node  corresponding  to  the  block  containing  the  first  statement  in  the  program  is  the  entry 
node  and  £my  node  having  no  successors  is  an  exit  node. 

The  test  data  selection  criteria  are  based  on  the  ways  in  which  values  are  associated  with 
variables,  and  how  these  associations  can  affect  the  execution  of  the  program.  This  aneilysis 
focuses  on  the  occurrences  of  variables  within  the  program. 

Data  flow  analysis  was  originally  used  for  compiler  optimization,  and  generally 
classifies  each  variable  occurrence  as  being  either  a  definition  or  a  use  [Sch73,  Hec77].  We 
distinguish  between  two  substantially  different  types  of  uses.  The  first  type  directly  affects 
the  computation  being  performed  or  allows  one  to  see  the  result  of  some  earlier  definition. 
We  call  such  a  use  a  computation  use  or  a  c-use.  Of  course,  a  c-use  may  indirectly  affect  the 
flow  of  control  through  the  program. 

In  contrast,  the  second  type  of  use  directly  affects  the  flow  of  control  through  the 
program,  and  thereby  may  indirectly  affect  the  computations  performed.  We  call  such  a  use 
a  predicate  use  or  p-use.  Occurrences  of  variables  in  the  language's  statements  are  thus 
classified  as  follows: 

1)  The  assignment  statement  "y  :=  <expression>"  where  the  variables  x^,  .  .  .  ,x„  occur 

in  the  expression  on  the  right-hand  side  contains  c-uses  of  jt; x^  followed  by  a 

definition  of  y. 

2)  The  input  statement  "  read  {x^,  .  .  .  ,x„)"  contains  definitions  oi x^.  .  .  .  .x„. 

3)  The  output  statement  "  write  (jc,,  .  .  .  ,jc„)"  contains  c-uses  of  j:,,  .  .  .  ,x„. 

4)  The  conditional  transfer  statement  "  if  <expression>  then  goto  <label>",  where  the 
variables  ;t.,  .  .  .  .x„  occur  in  the  expression,  contains  p-uses  of  j:;,  .  .  .  ,ar„. 

We  say  that  a  node  of  a  program  graph  contains  a  c-use  or  a  definition  of  a  variable  if 
there  is  a  statement  in  the  corresponding  block  containing,  respectively,  a  c-use  or  a 
definition  of  that  variable. 
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Since  we  are  interested  in  tracing  the  flow  of  data  betv^een  nodes,  any  definition  which 
is  used  only  within  the  node  in  which  that  definition  occurs  is  of  little  importance  to  us.  We 
thus  make  the  following  distinction:  a  c-use  of  a  variable  x  is  a  global  c-use  provided  there  is 
no  definition  of  x  preceding  the  c-use  within  the  block  in  which  it  ocoirs.  That  is,  the  value 
of  X  must  have  been  assigned  in  some  block  other  than  the  one  in  which  it  is  being  used. 
Otherwise  it  is  a  local  c-use.  Global  c-uses  are  often  called  locally  exposed  uses  in  the  data 
flow  analysis  literature  [Hec77]. 

A  conditional  transfer  statement  can  only  occur  as  the  last  statement  of  a  block.  It  has 
two  executional  successors,  which  are  always  in  two  different  blocks.  Since  the  values  of  the 
variables  occurring  in  the  predicate  portion  of  the  conditional  transfer  statement  directly 
determine  which  of  these  two  blocks  is  to  be  executed  next,  we  associate  p-uses  with  edges 
rather  than  with  the  node  in  which  the  predicate  portion  occurs.  If  the  final  statement  of  the 
block  corresponding  to  node  i  is  "  if  <expression>  then  goto  <label>",  where  the  variables 
Xi,  .  .  .  ,x„  occur  in  the  expression,  and  the  two  successors  of  node  i  are  nodes  j  and  k,  then 

we  will  say  that  edges  (i,j)  and  (i,k)  contain  p-uses  of  jc, x„.   Note  that  since  p-uses  are 

associated  with  edges,  no  distinction  need  be  made  between  local  and  global  p-uses. 

Let  X  be  a  variable  occurring  in  a  program.  A  path  (i,ny,...,n,„J)  m>0,  containing  no 
definitions  of  x  in  nodes  n^,...,n,„  is  called  a  definition  clear  path  with  respect  to  (wrt)  x  from 
node  i  to  node  j  and  from  node  i  to  edge  (n,„,j).  A  definition  of  a  variable  x  in  node  i  is  a 
non-local  definition  if  it  is  the  last  definition  of  x  occurring  in  the  block  associated  with  node  i. 
It  is  a  global  definition  if  it  is  a  non-local  definition  and  there  is  a  definition  clear  path  wrt  x 
from  i  to  either  a  node  containing  a  global  c-use  of  x  or  an  edge  containing  a  p-use  of  x. 
Thus,  a  global  definition  assigns  a  value  to  a  vsiriable  which  can  be  used  outside  the  node  in 
which  the  definition  occurs.  A  definition  which  is  non-local  but  not  global  can  never  be  used; 
if  the  program  contains  £my  such  definitions,  it  should  be  examined  for  possible  error. 

We  now  define  several  sets  which  will  be  needed  below  (see  figure  2.)  The  defuse 
graph  is  obtained  from  the  program  graph  by  associa':"i>g  a  set  with  each  edge  and  two  sets 
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def(i)       = 

{variables} 

{nodes} 

{edges} 

{x  €  V  1  X  has  a  non-local  definition  in  block  i} 

c-use(i)     = 
p-use(i,j)  = 
dcu(x,i)    = 

dpu(x,i)    = 

{x  €  V  1  X  has  a  global  c-use  in  block  i} 
{x  e  V  1  X  has  a  p-use  in  edge  (i,j)  } 
{j  €  N  1  X  e  c-use(j)  and  there  is  a 
def-clear  path  from  i  to  j} 
=    {(j,k)  e  E  1  X  €  p-use(j,k)  and  there 

is  a  def-clear  path  from  i  to  (j,k)  } 

Figure  2. 

with  each  node.  p-use(i,j)  is  the  set  of  variables  for  which  edge  (i,j)  contains  a  p-use.  def(i) 
is  the  set  of  variables  for  which  node  i  contains  a  non-local  definition,  c-use(i)  is  the  set  of 
variables  for  which  node  i  contains  a  global  c-use.  Let  i  be  a  node  and  let  x  f  def(i).  Then 
dcu(x,i)  is  the  set  of  all  nodes  j  such  that  x  has  a  global  c-use  in  node  j,  and  for  which  there  is 
a  definition  clear  path  wrt  x  from  i  to  j;  dpu(x.i)  is  the  set  of  all  edges  (j,k)  such  that  (j,k) 
contains  a  p-use  of  x,  and  for  which  there  is  a  definition  clear  path  wrt  x  from  i  to  j. 

We  now  informally  introduce  the  family  of  path  selection  criteria  which  are  formally 
defined  in  [Rap80]  and  [Rap82].  Let  G  be  a  program  graph,  and  P  be  a  set  of  paths  from  the 
unique  entry  node  of  G  to  the  unique  exit  node  of  G.  We  say  that  P  is  executed  if  every  path 
contained  in  P  is  traversed  during  the  course  of  executing  the  program  on  a  set  of  test  data. 
Then: 

•  P  satisfies  the  all-nodes  criterion  if  every  node  of  G  is  included  in  some  path  of  P. 

•  P  satisfies  the  all-edges  criterion  if  every  edge  of  G  is  included  in  some  path  of  P. 

•  P  satisfies  the  all-definitions  criterion  if  for  every  node  i  of  G  and  every  x  which  has  a 
global  definition  in  i,  P  includes  a  definition  clear  path  wrt  x  from  i  to  some  element  of 
dcu(x,i)  or  dpu(x,i).  That  is,  every  global  definition  must  be  used. 

•  P  satisfies  the  all-p-uses  criterion  if  for  every  node  i  and  every  x  which  has  a  global 
definition  in  i,  P  includes  a  definition  clear  path  wrt  x  from  i  to  all  elements  of  dpu(x,i). 
That  is,  P  includes  a  path  from  every  globai  definition  to  each  of  its  potential  p-uses. 


•  P  satisfies  the  all-c-uses/some-p-uses  criterion  if  for  every  node  i  and  every  x  which  has 
a  global  definition  in  i,  P  includes  some  definition  clear  path  wrt  x  from  i  to  every  node 
in  dcu(x,i);  if  dcu(x,i)  is  empty,  then  P  must  include  a  definition  clear  path  wrt  x  from  i 
to  some  edge  contained  in  dpu(x,i).  This  criterion  requires  that  every  global  c-use  of  a 
variable  x  globally  defined  in  node  i  must  be  included  in  some  path  of  P.  If  there  is  no 
such  c-use,  then  some  p-use  cf  the  definition  of  x  in  i  must  be  included. 

•  P  satisfies  the  all-p-uses/some-c-uses  criterion  if  for  every  node  i  and  every  x  which  has 
a  global  definition  in  i,  P  includes  a  definition  clear  path  wrt  x  from  i  to  all  elements  of 
dpu(x,i);  if  dpu(x,i)  is  empty,  then  P  must  include  a  definition  clear  path  wrt  x  from  i  to 
some  node  contained  in  dcu(x,i).  That  is,  every  p-use  of  each  variable  x  globally 
defined  in  node  i  must  be  included  in  some  path  of  P.  If  there  is  no  such  p-use,  then 
some  global  c-use  of  the  definition  of  x  in  i  must  be  included. 

•  P  satisfies  the  all-uses  criterion  if  for  every  node  i  and  every  x  which  has  a  global 
definition  in  i,  P  includes  a  definition  clear  path  wrt  x  from  i  to  all  elements  of  dcu(x,i) 
and  to  all  elements  of  dpu(x,i).  Thus,  P  must  include  a  path  from  every  global 
definition  to  each  of  its  uses. 

•  A  simple  path  is  one  in  which  cdl  nodes,  except  possibly  the  first  and  last,  are  distinct. 
A  loop-free  path  is  one  in  which  all  nodes,  are  distinct.  A  path  (nj^,...,^,,^^)  is  a  du-path 
with  respect  to  a  variable  x  if  n^  has  a  global  definition  of  x  and  either 

i)       n^i  has  a  c-use  of  x  and  {n^,...nj,ni^)  is  a  def-clear  simple  path  with  respect  to  x 

or 

ii)      (ny.Hi)  has  a  p-use  of  x  and  {n,,...,nj)  is  a  def-clear  loop-free  path  with  respect  to 

X. 

P  satisfies  the  all-du-paths  criterion  if  for  every  node  i  and  every  x  which  has  a  global 
definition  in  i,  P  includes  every  du-path  wrt  x.  Thus,  if  there  are  multiple  du-paths 
from  a  global  definition  to  a  given  use,  they  must  all  be  included  in  paths  of  P. 
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•  A  complete  path  is  a  path  [n-,...,n„)  such  that  n^  is  the  entry  node  and  n,„  is  an  exit 
node.  P  satisfies  the  all-paths  criterion  if  P  includes  every  complete  path  of  G.  Note 
that  programs  which  are  represented  by  graphs  containing  loops  may  contain  infinitely 
many  complete  paths. 

We  call  the  criteria  all-definitions,  eill-p-uses,  all-c-uses/some-p-uses,  all-p-uses/some-c- 
uses,  all-uses,  and  all-du-paths  dataflow  criteria. 

We  say  that  criterion  c^  includes  criterion  Ci  if  for  every  program  graph  G,  any  set  of 
paths  from  the  entry  node  of  G  to  the  exit  node  of  G  that  satisfies  c^,  also  satisfies  c^.  We 
say  that  c^  strictly  includes  Ci,  denoted  c^  =>  Ci  provided  that  c^  includes  C;,  and  for  some  G 
there  is  a  set  of  paths  of  G  that  satisfies  c,  but  not  c^.   The  following  is  proved  in  [Rap80]: 

Theorem: 

The  family  of  criteria  is  partially  ordered  by  strict  inclusion  as  shown  in  figure  3. 
Furthermore,  criterion  c,  strictly  includes  aiterion  Cj,  if  and  only  if  it  is  explicitly  shown 
to  be  so  in  figure  3  or  follows  from  the  transitivity  of  the  relationship. 

3.   Design  and  Implementation  of  ASSET 

ASSET  takes  as  input  a  program  called  the  subject  program,  written  in  the  subset  of 
Pascal  described  above,  one  of  the  data  flow  adequacy  criteria,  and  a  set  of  test  data.  It  is 
assumed  that  the  program  is  syntactically  correct  and  that 

i)       If  L  is  the  target  of  some  goto  then  there  is  exactly  one  labeled  statement  with  label  L. 

ii)      All  transfers  are  effective,  i.e.    a  statement  goto  L  cannot  immediately  physically 

precede  the  statement  labeled  L. 
iii)     There  are  no  syntactically  endless  loops,  i.e.  on  every  path  (n,,...,n^,n)  there  is  some 

block  containing  a  conditional  transfer  to  a  block  which  is  not  on  the  path, 
iv)     There  is  no  syntactically  unreachable  code,  i.e.  there  are  paths  from  the  start  node  to 

every  node  in  the  flow  graph. 
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ALL-PATHS 


N 


ALL-DU-PATHS 


\ 


V 


ALL-USES 


ALL-C-USES  / 
SOME-P-USES 


ALL-P-USES  / 

SOME-C-USES 
/ . 


ALL-P-USES 


ALL-EDGES 


ALL-NODES 


Figure  3. 

ASSET  checks  that  these  conditions  are  met  and  flags  any  violations. 

ASSET'S  first  step  is  the  production  of  the  subject  program's  def-use  graph  (see  figure 
4).  On  the  first  pass  through  the  SLifcject  program,  a  table  of  label  occurrences  in  labeled 
statements    and   their   uses   in   transfer   statements   is   constructed.     On   the   second   pass, 
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statements  are  classified  as  labeled  or  unlabeled,  and  as  conditional  transfer,  unconditional 
transfer,  or  other,  and  this  information,  along  with  the  label  table,  is  used  to  divide  the 
program  into  blocks  and  tc  produce  the  flow  graph,  which  is  represented  as  an  array  of 
adjacency  lists. 

Variables  are  inserted  into  a  symbol  table  called  the  def-use-table.  Variable  occurrences 
are  classified  as  c-use,  p-use,  or  definition  (see  section  2)  according  to  whether  they  appear 
on  the  right  hand  side  of  an  assignment  statement  or  in  a  write  statement,  in  a  conditional 
transfer  statement,  or  on  the  left-hand  side  of  an  assignment  statement  or  in  a  read 
statement.  In  each  block  the  global  c-uses  and  non-local  definitions  are  distinguished  from 
the  local  ones  according  to  position  within  the  block.  The  global  c-uses,  p-uses,  and  non- 
local definitions  (which  we  will  refer  to  as  c-uses,  p-uses,  and  definitions)  are  recorded  in  the 
def-use-table.   The  flow  graph  along  with  the  def-use-table  constitute  the  def-use  graph. 

The  flowgraph  provides  the  information  needed  to  insert  probes  in  the  subject  program. 
A  statement  of  the  form  writein  (traversed,  <  block  number >)  is  inserted  into  each  block. 
The  modified  subject  program  is  then  compiled  by  a  Pascal  compiler  and  executed  on  each 
element  of  the  test  set.  A  record  of  each  test  is  written,  including  the  input,  the  output,  and 
the  path  traversed.  These  results  can  be  examined  by  the  person  testing  the  program  to 
determine  whether  the  test  was  successful,  i.e.  whether  the  program  met  its  specification  on 
each  test  datum.  The  paths  executed  by  the  various  test  data  are  recorded  in  the  file 
"traversed",  separated  by  markers.  These  will  be  used  later  to  determine  whether  or  not  the 
test  fulfilled  the  given  adequacy  criterion. 

ASSET  uses  the  def-use  graph  to  determine  which  pairs  or  paths  are  required  by  the 
given  criterion.  For  the  criteria  all-definitions,  all-p-uses,  all-c-uses/some-p-uses,  all-p- 
uses/some-c-uses  and  all-uses,  this  entails  constructing  the  sets  dcu(x,i)  and  dpu(x,i),  defined 
above.  These  sets  are  constructed  by  performing  a  series  of  depth-first  searches,  one  for 
each  non-local  definition  of  a  variable.  To  construct  dcu(x,i)  and  dpu(x,i)  a  depth-first 
search  is  performed  beginning  at  node  i.    Each  Timr  a  node  j  is  visited,  the  def-use-table  is 
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checked  to  determine  whether  x  (.  c-use(j),  x  ?  p-use(j),  or  x  €  def(j).  If  x  e  c-use(j)  [resp. 
p-use(j)]  then  j  is  added  to  dcu(x,i)  [resp.  dpu(x,i)].  If  x  C  def(j)  then  the  search  backtracks; 
otherwise  it  continues.  In  effect,  the  search  visits  every  node  which  is  syntactically  reachable 
from  i  by  some  definition-clear  path.  After  each  search,  the  sets  dcu(x,i)  and  dpu(x,i)  are 
recorded  in  a  file.  These  sets  will  be  used  later  to  determine  whether  the  paths  executed  by 
the  test  data  include  the  required  def-use  pairs. 

This  series  of  searches  takes  time  0(N^V)  in  the  worst  case,  where  N=  number  of 
blocks  and  V  =  number  of  variables,  since  at  most  N\'  depth-first  searches  are  performed, 
each  taking  time  0(N)  because  the  graphs  involved  are  sparse.  In  fact,  since  only  those 
nodes  reachable  along  a  definition-clear  path  from  node  i  are  visited  on  a  search  starting  at  i, 
the  worst  case  will  not  usually  occur. 

If  the  criterion  is  all-du-paths,  then  for  each  definition  of  variable  x  in  node  i,  ASSET 
explores  the  graph  in  a  depth-first  manner,  recording  in  a  file  every  du-path  from  i  to  a  node 
containing  a  c-use  or  an  edge  contEiining  a  p-use  of  x.  Since  there  are  fl(2  )  du-paths,  each 
of  which  must  be  explicitly  visited,  this  portion  of  the  algorithm  has  an  exponential  lower 
bound  on  worst-case  behavior. 

ASSET  next  determines  which,  if  any,  of  the  pairs  or  paths  required  by  the  given 
adequacy  criterion  were  not  executed  by  the  given  test  data.  This  is  done  by  treating  paths  as 
strings  on  the  alphabet  A  =  {node  numbers},  expressing  the  required  conditions  in  terms  of 
regular  expressions  on  A,  and  running  the  paths  traversed  by  the  test  data  through 
appropriate  acceptors. 

To  handle  the  criteria  all-definitions,  all-p-uses,  all-c-uses/some-p-uses,  all-p-uses/some- 
c-uses,  all-uses,  observe  that  the  set  of  complete  paths  which  include  definition-clear  subpaths 
from  node  i  to  node  j  can  be  described  by  the  regular  expression 

A*i(A  -  D)'jA* 
where  D  -  {  nodes  k  |  x  e  def(k)  }.   Those  including  definition-clear  subpaths  from  i  to  edge 
(j,k)  can  be  described  by 
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A*i(A  -  D)*jkA* 

ASSET  simulates  the  parallel  execution  of  a  family  of  deterministic  finite  automata,  one 
dfa  for  each  def-use  pair,  as  indicated  by  the  algorithm  in  figure  5. 


type  automaton  =  record 

variable:  hashrange;     {index  of  defined  variable 

in  def-use-table} 
def ,  {node  in  which  def  occurs} 

use   :  node  number;     {node  in  which  use  occurs} 
state:  (ql,q2,qf); 

accepted:  boolean;         {initially  false} 
end; 

var  c-use, 

p-use  :  array  [1..  max  #  uses]  of  automaton; 

{This  algorithm  simulates  the  execution  of  the  automata  which 
recognize  def-c-use  associations.  The  algorithm  for  def -p-use 
associations  is  similar.  } 

initialize; 

repeat      {process  next  node  in  input  string} 
read(node); 

for  i:=  1  to  max  #  uses  do 
{update  state  of  /'''  automaton  } 
with  c-use[i]  do 
case  state  of 

ql:   If  node  =  def  then  state  :=  q2; 
q2:  if  node  =  use  then 
begin 
state  :=  qf; 
accepted  :=  true 
end 
else  if  variable  d  def  (node)  then  state  :=  ql; 
qf:   {null}; 

end   {case} 
until  end-of-path; 


Figure  5. 

After  running  these  automata  on  all  of  the  paths  traversed  by  test  data,  it  is  easy  to 

determine    which    of   the    CTiteria    all-definitions,    all-p-uses,    all-c-uses/sorae-p-uses,    all-p- 
uses/some-c-uses  and  all-uses  have  been  fulfilled  by  examining  the  arrays  p-use  and  c-use. 

In  order  to  determine  which,  if  any,  of  the  paths  required  by  the  all-du-paths  criterion 
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have  not  been  exercised  by  the  test  data,  ASSET  uses  the  fact  that  a  traversed  path,  p, 
exercises  the  du-path  q  if  and  only  if  p  is  of  the  form  A'qA*.  ASSET  uses  a  pattern 
matching  algorithm  to  determine  which  of  the  required  du-paths  appear  as  subpaths  of  some 
traversed  path. 

4.   Uses  of  ASSET 

ASSET  has  many  uses,  both  practical  and  experimental.  It  can  be  used  to  evaluate  test 
data,  to  aid  in  the  selection  of  test  data,  to  enhance  program  reliability  through  data-flow 
anomaly  detection,  to  aid  in  program  debugging,  to  augment  program  doamientation,  emd  to 
gather  information  on  the  cost  and  effectiveness  of  data  flow  testing. 

Given  a  subject  program,  a  test  set,  and  one  of  the  data  flow  test  data  adequacy  criteria, 
ASSET  determines  whether  the  test  set  is  adequate  with  respect  to  the  criterion.  In  addition, 
it  produces  a  list  of  those  pairs  of  nodes  (or  in  the  case  of  the  all-du-paths  criterion,  paths),  if 
any,  which  are  required  by  the  criterion,  but  not  exercised  by  the  test  data.  The  person 
testing  the  program  can  then  use  this  list  to  guide  the  selection  of  additional  test  data.  Of 
course,  running  ASSET  with  the  empty  test  set  as  input,  provides  guidance  for  the  initial 
selection  of  test  data. 

While  constructing  the  def-use  graph  and  the  lists  of  pairs  or  paths  required,  ASSET 
looks  for  data  flow  anomalies  which  may  indicate  the  presence  of  errors.  In  [Ost76], 
Osterweil  and  Fosdick  point  out  that  along  any  complete  path  the  following  "rule"  is  expected 
to  be  obeyed:  "A  definition  must  be  followed  by  a  reference,  before  another  definition  or 
undefinition."  They  call  a  violation  of  this  rule  a  type  2  data  flow  anomaly.  Such  a  data  flow 
anomaly  is  not  necessarily  an  error,  but  often  indicates  the  presence  of  an  error,  such  as  a 
misspelled  variable  name. 

ASSET  detects  such  data  flow  anomalies  under  certain  circumstances.  Any  type  2 
anomaly  occuring  within  a  single  block  is  detected  while  the  def-use  graph  is  being 
constructed.    During  the  construction  of  the  sets  dcu(x,i)  and  dpu(x,i),  all  occurrences  of  a 
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definition  which  is  never  used  and  some  (but  not  necessarily  all)  occurrences  of  two 
definitions  of  the  same  variable  without  an  intervening  use  are  detected.  The  nature  smd 
location  of  any  anomalies  detected  are  reported  to  the  user,  who  should  then  examine  the 
relevent  portions  of  the  program  for  possible  error. 

ASSET  also  serves  as  a  debugging  aid.  If  the  program  outputs  the  wrong  value  for 
variable  x  on  test  case  t,  the  programmer  can  use  information  about  the  path  executed  by  t 
and  the  def-use  associations  for  x  to  help  focus  the  search  for  the  bug.  ASSET  can  be 
modified  so  that  it  can  serve  as  an  interactive  debugging  aid.  This  can  be  done  by  modifying 
the  probes  which  are  inserted  into  the  subject  program  so  that  in  addition  to  recording  the 
node  number  on  the  file  "traversed",  they  would  serve  as  optional  stops  at  which  the  values 
of  any  variables  having  non-local  definitions  or  uses  in  the  block  could  be  reported  to  the 
user. 

ASSET  produces  several  useful  documents  pertaining  to  the  subject  program.  Graphical 
output  of  the  flow  graph  makes  it  easier  to  read  and  understand  the  program.  The  file 
containing  the  inputs,  outputs,  and  paths  traversed  by  the  test  data  provides  a  well  organized 
record  of  the  testing  session.  This  can  be  useful  if  a  modified  version  of  the  program  is  to  be 
tested  or  if  the  test  data  is  to  be  subjected  to  another  adequacy  criterion.  ASSET  can  easily 
be  modified  so  as  to  include  other  information,  such  as  performance  statistics  in  this  file. 

We  plan  to  use  ASSET  to  do  empirical  studies  of  the  cost  and  effectiveness  of  the  data 
flow  criteria.  Weyuker  [Wey84]  has  shown  that  for  a  program  with  n  variables,  m 
assignment  statements,  i  input  statements,  and  t  conditional  transfer  statements,  the  all-nodes 
and  all  edges  criteria  require  at  most  t+1  test  cases;  all-definitions  requires  at  most  ra-l-ixn 
test  cases;  The  all-p-uses,  all-c-uses/some-p-uses,  all-p-uses/some  c-uses,  and  all  uses  criteria 

require  at  most test  cases.   All-du-paths  requires  at  most  2  test  cases.   In  each  case 

there  exist  programs  requiring  that  number  of  test  cases.  However  the  types  of  programs 
which  require  these  upper  bounds  are  peculiar.  We  plan  to  use  ASSET  to  obtain  empirical 
results  indicating  the  number  of  test  cases  required  by  "real"  programs.   The  effectiveness  of 
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the  various  criteria  can  be  investigated  by  using  ASSET  on  programs  with  known  errors  and 
attempting  to  categorize  which  errors  each  criterion  is  likely  to  detect. 

5.   Conclusions  and  Future  Work 

We  have  described  the  design  and  implementation  of  a  tool  based  on  the  theory 
developed  in  [Rap80,Rap82].  ASSET  uses  data  flow  information  to  aid  in  the  selection  and 
evaluation  of  test  data  for  programs  written  in  a  small  subset  of  Pascal.  It  also  enhances 
program  reliability  by  performing  data  flow  anomtily  detection,  by  aiding  in  debugging  and 
by  providing  useful  documentation.  The  utility  of  ASSET  is  limited  by  the  fact  that  subject 
programs  must  be  written  in  an  extremely  simple  language  which  permits  only  simple 
variables  and  elementary  control  flow  mechanisms  (sequential  flow,  conditional  and 
unconditional  transfers.) 

We  plan  to  extend  the  theory  developed  in  [Rap80,Rap82,Wey84]  to  cover  a  more 
sophisticated  programming  language,  including  such  features  as  arrays,  dynamic  allocation, 
aliasing,  procedure  and  function  calls,  and  recursion.  The  tool  described  in  this  paper  can 
then  be  used  as  a  prototype  for  a  tool  which  aids  in  selection  and  evaluation  of  test  data  for 
programs  written  in  such  a  language.  We  expect  that  the  upgraded  tool  will  prove  to  be  an 
effective,  reasonable-cost  aid  to  significanUy  enhancing  software  reliability. 
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ABSTRACT 

This  paper  describes  ASSET,  a  tool  which  uses  information  about  a 
program's  data  flow  to  aid  in  selecting  test  data  for  the  program  and  to 
evaluate  test  data  adequacy.  ASSET  is  based  on  the  family  of  data  flow  test 
selection  and  test  data  adequacy  criteria  developed  by  Rapps  and  Weyuker. 
ASSET  accepts  as  input  a  program  written  in  a  subset  of  Pascal,  a  set  of  test 
data,  and  one  of  the  data  flow  adequacy  criterion  and  indicates  to  what  extent 
the  criterion  has  been  satisfied  by  the  test  data. 
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